The invention relates generally to the area of gesture-based user interfaces, and more specifically to the creation of grammars for gesture-based user interfaces, particularly in the context of touch-based user interfaces.
Until recent years the dominant form of Graphical User Interface (GUI) model for general-purpose computers has been (initially) the Direct Manipulation and Desktop Metaphor (see for example http://en.wikipedia.org/wiki/Direct_manipulation), often attributed to B. Shneiderman in 1983[1], and later their arguable descendent WIMP (“Window, Icon, Menu, Pointer/Pointing/Pull-Down/Pop-up”) GUI (see for example http://en.wikipedia.org/wiki/History_of_the_graphical_user_interface and http://en.wikipedia.org/wiki/WIMP_(computing)). Many additional user interface mechanisms have been explored, and many of these (for example, speech recognition) map directly into the Direct Manipulation and Desktop Metaphor paradigm. The pointing devices employed notably include not only the computer mouse but a number of surrogate forms emulating the mouse metaphor, namely various trackballs, keyboard-sticks, touch-screens, and touchpads (including the KoalaPad™ in 1984—see for example http://en.wikipedia.org/wiki/Koala_Pad). These touch-based computer interfaces (touch-screens and touchpads) indeed operated as mere stand-in emulations of computer mouse functionality.
It is noted that, prior to computer touch-screens and touchpads various elevator, machine, and appliance controls from the 1950's (and likely earlier) included touch-operated on-off switches, and various 1970's music synthesizers included touch-keyboards and one-dimensional touch “ribbon controllers.”
Work on more sophisticated touch-based computer and control interfaces that accommodate and utilize touch-based gestures has a long history, some of it widely recognized (for example http:/www.billbuxton.com/multi-touchOverview.htm) and less well-known such as the High Dimensional Touch Pad (HDTP) technology represented for example by (1999 priority date) U.S. Pat. No. 6,570,078, U.S. patent application Ser. No. 11/761,978, U.S. patent application Ser. No. 12/418,605, and some at least two dozen other related pending patent applications. The most well-known work is that of Wayne Westerman and his thesis professor John Elias. The approach that work took to touch-based gestures has since been incorporated into in a large number of Apple™ products, and subsequently widely adopted by large a number of other handheld, tablet, laptop, and other computing-based devices made by many product manufacturers.
Within this period of time there was a considerable amount of work and product relating to pen/stylus-based handwriting interfaces (see for example http://en.wikipedia.org/wiki/Pen_computing), some including a few early gesture capabilities (http://en.wikipedia.org/wiki/Pen_computing#Gesture_recognition).
More recently video-camera-based free-space hand-gesture input have appeared, It is noted that (1999 priority date) U.S. Pat. No. 6,570,078 teaches use of a video camera as an input device to deliver HDTP capabiities extended to free-space hand-gesture input.
Although the widely adopted approach to gesture-based multi-touch user interfaces developed by Westerman and Apple has become pervasive and extends the WIMP GUI operations to include a number of allegedly “new” metaphor-based specialty operations (such as “swipe,” “stretch,” “pinch,” “rotate,” etc), that approach is hardly the last word in touch-based user interfaces. The HDTP approach to touch-based user interfaces, represented for example by represented for example by U.S. Pat. No. 6,570,078, U.S. patent application Ser. No. 11/761,978, U.S. patent application Ser. No. 12/418,605, provides a framework that includes or supports today's widely adopted gesture-based multi-touch user interface features and further supports a wide range of additional capabilities which transcend and depart from today's widely adopted gesture-based multi-touch user interfaces.
A first aspect of the HDTP approach includes the capability for deriving more than the two-dimensional ‘continuous-adjustment’ user inputs than are provided by today's widely adopted gesture-based multi-touch user interface “geometric location” operations (such as X-Y location, “flick” X-Y location-change velocity, “flick” X-Y location-change angle). For example the HDTP approach to touch-based user interfaces can provide additional ‘continuous-adjustment’ user inputs such as:                Yaw-angle of a contacting finger, thumb, palm, wrist, etc.;        Roll-angle of a contacting finger, thumb, palm, wrist, etc.;        Pitch-angle of a contacting finger, thumb, palm, wrist, etc.;        Downward pressure of a contacting finger;        Spread angle between each pair of contacting finger(s), thumb, palm, wrist, etc.;        Differences in X location between each pair of contacting finger(s), thumb, palm, wrist, etc.;        Differences in Y location between each pair of contacting finger(s), thumb, palm, wrist, etc.;        Differences in downward pressure between each pair of contacting finger(s), thumb, palm, wrist, etc.;        Rates-of-change for the above.        
These additional capabilities widely expand the number and types of gestural, geometric, and spatial-operation metaphors that can be provided by touch interfaces. Further, various types of conditional tests may be imposed on these additional ‘continuous-adjustment’ inputs, permitting productions of and associations with symbols, domains, modalities, etc.
Today's widely adopted gesture-based multi-touch user interfaces recognize the number of multiple-touch contacts with the touch interface surface. A second aspect of the HDTP approach to touch-based user interfaces are additional ‘shape’ user input recognitions distinguishing among parts of the hand such as:                Finger-tip;        Finger-joint;        Flat-finger;        Thumb;        Cuff;        Wrist;        Palm;        Left-hand;        Right-hand.        
Today's widely adopted gesture-based multi-touch user interfaces recognize individual isolated gestures. A third aspect of the HDTP approach to touch-based user interfaces can provide yet other additional features such as:                Compound touch gestures;        Attributes of individual component elements comprised by a gesture such as:                    Order of individual component element rendering;            Relative location of individual component element rendering;            Embellishment in individual component element rendering (angle of rendering, initiating curve, terminating curve, intra-rendering curve, rates of rendering aspects, etc.);                        Connected gestures;        Context-based interpretation/action/semantics;        Inheritance-based interpretation/action/semantics;        Syntactic grammars.The present patent application, along with other associated co-pending U.S. patent cited herein, directs further attention to these topics, both in the context of HDTP technology as well as other user interface technologies including:        Simple touch user interface systems found in handheld devices, laptops, and other mobile devices        Video camera-based free-space gesture user interface systems        
In the case of the HDTP approach to touch-based user interfaces, these provide the basis for                (1) a dense, intermixed quantity-rich/symbol-rich/metaphor-rich information flux capable of significant human-machine information-transfer rates and        (2) an unprecedented range of natural gestural metaphor support.The latter (1) and its synergy with the former (2) is especially noteworthy, emphasized the quote from the recent cover story in the February 2011 Communications of the ACM [2]:        “Gestures are useful for computer interaction since they are the most primary and expressive form of human communication.”        
The next-generation user interface work in academia, as well as in video games, however, is now directing attention to video-camera-based free-space gesture input, owing great debts to the pioneering experiential/installation/performance-art-oriented real-time video-based computer control work of Myron Kruger. These camera-based free-space gesture input user interfaces will be providing a range of possibilities comprising, at least tabula rasa, ranges and possibilities not unlike those provided by the HDTP approach to touch-based user interfaces. (In fact (1999 priority date) U.S. Pat. No. 6,570,078, U.S. patent application Ser. No. 11/761,978 teach use of one or more video cameras as alternative input sensors to HDTP processing so as to respond to free-space hand gestures.)
However, it is not at this time clear whether the camera-based free-space gesture input user interface community will see these opportunities or simply incrementally adapt and build on WIMP frameworks, the Westerman/Apple approach, 3D extrapolations of desktops, etc. Additionally, these camera-based free-space gesture input user interface approaches have their own usage challenges (not the least of which including arm fatigue, input on/off detection (“Midas Touch problem”) and computation challenges if trying to adopt rich-semantic inputs (for example, recognitions of ASL and other sign languages remains computationally out or reach even well-funded research labs loaded with computers [2]).
It is believed this effort, in addition to the role it provides to contemporary touch interfaces and HDTP technology, could deliver potential utility to next-generation touch interfaces and provide a framework and an example perhaps of possible value to the camera-based free-space gesture input user interface community as the possibilities and opportunities for camera-based free-space gesture input user interface technology and its applications are explored, developed, and formalized.